HYPERBOLIC ALEXANDROV-FENCHEL QUERMASSINTEGRAL 

INEQUALITIES. I 
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Abstract. In this paper we prove the following geometric inequality in the hyperbolic space 
H n (n > 5), which is a hyperbolic Alexandrov-Fenchel inequality, 

, r ) I |S| \* / ISI 



£ I \OJ n —l J \U) n — 1 

provided that E is a horospherical convex hypersurface. Equality holds if and only if E is a 
geodesic sphere in H n . 



1. Introduction 

The Alexandrov-Fenchel inequalities for quermassintegrals of convex domains in M n , as a gen- 
eralization of the classical isoperimetric inequality, play an important role in classical geometry. 
For a bounded smooth domain f2 C M. n with boundary d£l = S, let k = (k±, K2, ■ ■ ■ , n n -i) be the 
set of the principal curvatures of £ and : M n_1 — > R the k-th elementary symmetric function. 
One of equivalent definitions of the quermassintegrals of f2 is 

(1-1) Ki-fc(fi) = Cn,k J o"fc-l(«), k > 1, 

where c n ^ = C^„ 1 /C^l}. In this paper we denote C^_ 1 = k^-i-ky. • ^™ * s ^ ne vomme °f ^ U P to 
a constant multiple and the area of S. The celebrated Alexandrov-Fenchel quermassintegral 
inequalities state that if Q is convex, then for < i < j < n, 

(L2) v n -j(n)^ > v n -i(p.)^ 



Vn-jiB)—! V n .i{B)—i 

When i = and j = 1, (jl.2p is the isoperimetric inequality. For A; > 1 in (jl.ip . inequality (|1 .2j) 
is equivalent to 

-J — f aA , 0<j<k<n—l, 

where w n _i is the area of the standard sphere S n_1 . (In this paper we use a convention that 
o"o = 1.) Hence inequality (|1.3p is also interpreted as a generalization of the isoperimetric 
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inequality. When k = 1, this is usually called a Minkowski inequality. When j = 0, it is 



(1.4) / a k > Cti^n-i 

The Alexandrov-Fenchel quermassintegral inequalities in M. n have been intensively studied in the 
last several decades. See classical books of Santos[39j, Burago-Zalgaller [9J and Schneider [H] . 
For the non-convex domains, see the recent interesting work of Guan-Li [25j and Huisken [31] 
and also the work of Chang- Wang [11]. Here we just mention a conformal version of Alexandrov- 
Fenchel quermassintegral inequalities in E> n in [29], which is curious that there is a very closed 
relation to the hyperbolic Alexandrov-Fenchel quermassintegral inequalities which we want to 
establish in this paper. 

In this paper we are interested in its analogue in the hyperbolic space H n . Let H n = M + x S" _1 
with the hyperbolic metric 

g = dr 2 + sinh 2 rgg n -i, 

where g§n-i is the standard round metric on the unit sphere S n_1 . The isoperimetric inequality 
on the hyperbolic space H n was obtained by Schmidt [30]. See a flow approach in [35]. When 
n = 2, the hyperbolic isoperimetric inequality is 

L 2 > ^ttA + A 2 , 

where L is the length of a curve 7 in H 2 and A is the area of the enclosed domain by 7. 
Moreover, equality holds if and only if 7 is a circle. There are many attempts to establish 
Alexandrov-Fenchel inequalities on the hyperbolic space M n . See, for example, [38] and [12]. In 
|18j . Gallego-Solanes proved by using integral geometry the following interesting inequality for 
convex domains in HP, precisely, there holds, 



(1.5) / a k dn > cC*_ x \Y> 



'S 

where c = 1 if k > 1 and c = (n — 2)/(n — 1) if k = 1 and |S| is the area of X. Here d\x is the 
area element of the induced metric. See also pjj. The above inequality (jl.5|l (k > 1) is sharp in 
the sense that the constant c could not be improved. However, this inequality is far away from 
being optimal, especially when |S| is small. 

Recently motivated by the study of the quasi-local mass and the Penrose inequality, Brendle- 
Hung-Wang [8] established the following Minkowski type inequalities (i.e., k = 1) 

(1.6) J [X'H -(n-l)(VA',i/)J^ > (n-l)w^|S|^r, 

and de Lima and Girao [15] proved the following related inequality 

f , ( , ISI "-2 IEI , 2 \ 

(1.7) / X'Hdfi > (n - l)w n _! (-L-L)^ + (_U_) n-i f 

JT, V w n-l W n _i / 

where A'(r) = coshr, if S is star-shaped and mean convex (i.e. H > 0). This total mean 
curvature integral with the weight A' appears naturally in the definition of the Brown- York 
mass and the Liu-Yau mass and in the Penrose inequality for asymptotically hyperbolic graphs 
|14j . The higher order mean curvature integrals J A'o^fc-i appear also in our work [21] on a 
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new mass on asymptotically hyperbolic graphs. It is an interesting open problem if one can 
generalize (|1.6p and (I1.7P for general k. 

In this paper we are interested in the Alexandrov-Fenchel quermassintegral inequalities in H n 
for curvature integrals without the weight A', i.e, for 

a k d\i. 

s 

Such curvature integrals, like in the Euclidean case, have a close relationship with quermassin- 
tegral in M n . See for example |39 t 118 1 1^1- We will give more details in our forthcoming paper 
|22j . Such an inequality for 02 was first proved by Li-Wei-Xiong in a recent work 

/ x f , (n-l)(n-2) /, . 2 _ 3 

(1.8) h a2d ^- — 2 — [\n+<--\\n~^ 

provided that E C H™ is a star-shaped and two-convex hypersurface, ie., a\ > and 02 > 0. 
In this paper we obtain 

Theorem 1.1. Let n > 5. IfHc M n is horospherical convex, then 

f 1 1 ra-5 \ 2 

'El \ 2 / IE! \ 2~ 



(1.9) / M/^ > C£-iW„_i 4 ^ + 

where 0J n -\ is the area of the unit sphere S n_1 and |E| is i/ie area o/E. Equality holds if and 
only if E is a geodesic sphere. Moreover, when n = 5, $1.9\) holds provided that E C M n is a 
star-shaped and two-convex hypersurface, ie., o~\ > and 02 > 0. 

Theorem 11.11 implies trivially an (equivalent) isoperimetric type result: In the class of horo- 
spherical convex hypersurfaces with fixed area, the minimum of J a^d[i is achieved by and only 
by geodesic spheres. 

Unlike in the case of W 1 , we believe that in the case of IH n the parity of k in the fc-scalar 
curvature o~k plays a role in the Alexandrov-Fenchel quermassintegral inequalities^. Namely, for 
odd k the Alexandrov-Fenchel quermassintegral inequalities should look like (11.6|) or (II. 7p , while 
for even k (II. 8j) and (II. 9h are the correct and the best ones. See also Theorem 13.71 below. In a 
forthcoming paper [22] we will establish the Alexandrov-Fenchel quermassintegral inequalities 
for general even k. 

E C H n is horospherical convex if all principal curvatures are larger than or equal to 1. The 
horospherical convexity is a natural geometric concept, which is equivalent to the geometric 
convexity in Riemannian manifolds. Through our work, we believe that it is (almost) the best 
condition for inequality (II. 9p . See Remark 13.41 and Remark 13.51 below. 

The fundamental idea to show the above geometric inequalities is the same: Consider a 
suitable functional and a suitable geometric flow and prove this functional is non-increasing under 
the geometric flow. If the flow converges to the standard sphere, then we have an inequality, 
with a best constant achieved by the standard sphere. The flow we use is the inverse curvature 
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flow studied by Gerhardt [23] 

(1-10) -7T = ; V, 

y ' dt 4 CJ4 

where v is the outer normal of E. 

The first problem we meet is: what is the suitable functional for our inequality (jl.9p ? By the 
work of Brendle-Hung-Wang [8], de Lima-Girao |15j and Li-Wei-Xiong [32], one may guess that 
the following functional 

M-m n(r\ \rm f J (n-3)(n-4) ( n - l)(n - 2){n - 3)(n - 4) 
(1.11) Q(£) := |£| y s g cr 2 + — 

could be a good candidate. In fact, 

/ (n-3)(n-4) (n - l)(n - 2)(n - 3)(n - 4) 
/ 2 := ^ 4 - a 2 + 

is the Gauss-Bonnet curvature L 2 up to a constant multiple. The Gauss-Bonnet curvature L 2 is 
an intrinsic geometric invariant which is a natural generalization of the scalar curvature R. For 
the Gauss-Bonnet curvature L 2 , see for example |20j . We remark that the functional considered 
in [31] is in fact the Yamabe quotient for the scalar curvature R. Our functional (jl.lip is also a 
Yamabe type quotient for L 2 . Due to the complication of the geometry of the hyperbolic space, 
we obtain a variation formula of f l 2 , which has three terms that we have to deal with. Unlike 
the cases of proving inequalities (|1.6[> . (|1.7p and (jl.8p . one can not directly use the Newton- 
MacLaurin inequalities (Lemma I2.ip to deal with these three terms. More precisely, among the 
three terms appeared in (|3.ip . 



a 5 a 3 4(n-5) 4(n - 3) cxf 

5 — ct 4 , — c7 2 -3 — , 

CJ4 n — 4 n — 4 04 



are non-positive, and the term 



0-103 4(n - 1) 



04 n — 4 

is non-negative in the use of the Newton-MacLaurin inequalities. For the monotonicity of the 
functional Q we need to show that the sum of these 3 term is non-positive. Hence we have to 
deal with them together. Since these terms have different scaling of k, one could not expect 
the sum of these three terms is non-positive for all k = (k%, ■ ■ ■ , n n —i ) € Fortunately 
we show that it does be non-positive, if Ki > 1 for all i. See Proposition 13.31 This is one 
of crucial points of this paper, where the assumption of the horospherical convexity plays a 
crucial role. By applying the work of Gerhardt on the inverse curvature flow, one can show that 
flow (jl.lOp preserves the condition of the horospherical convexity. Therefore, the functional Q 
defined in (jl.lip is non- increasing under flow (jl.lOp . Hence, in order to obtain an inequality we 
now only need to consider the limit of Q under flow (jl.lOp . Now, we meet another problem, 
flow (jl.lOp converges only asymptotically in the sense presented in Proposition 2.2, namely the 
flow converges asymptotically to a sphere with a metric g conformal to the standard round 
metric on S n_1 . We show that along the flow, the induced metric has (asymptotically) positive 
Schouten tensor, if the evolving hypersurface is horospherical convex. For such a metric on S n ~ 1 , 
a generalized Sobolev inequality was proved by Guan-Wang [29J in conformal geometry. Here 
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the horospherical convexity plays again an important role. With this inequality we get a best 
estimate for Q in Theorem 13,71 Therefore Theorem 11.11 follows . It is interesting to see that the 
results in conformal geometry on the sphere are closely related to the hyperbolic Alexandrov- 
Fenchel inequality in HP\ The connecting bridge is the inverse curvature flow of Gerhardt |23j . 
See also the work of Ding [16] . For the recent related work see O [32] . 

The rest of this paper is organized as follows. In Section 2 we present some basic facts about 
the elementary functions a^, the variational formula for J and recall a generalized Sobolev 
inequality from |29j . The preservation of the horospherical convexity under a inverse curvature 
flow considered by Gerhardt, together with its convergence, is given in this section. In Section 
3, we prove the crucial monotonicity of Q, analyze its asymptotic behavior under flow (jl.lOp . 
and prove Theorem II .li 



2. Preliminaries 

Let dfc be the k-th. elementary symmetry function Ok '■ W l ~ l — > R defined by 

a k (A)= X ii--- X h for A = (Ai,-" ,A„_i) S ra 

h<---<ik 



The definition of can be easily extended to the set of all symmetric matrix. The Garding 
cone r^" is defined as 

T+ = {A G R n_1 I aj(k) > 0, Vj < k}. 
We collect the basic facts about a^, which will be directly used in this paper. For other related 
facts, see a survey of Guan [23] or [34] . 

Lemma 2.1. For A € , we have the following Newton- MacLaurin inequalities 
^ 21 ^ Qfc-iQfc+i < k(n-k-l) 



a 



k 



(jfe + l)(n-Jfe)' 



(2.2) aiak - 1 > Hn ~ 1} 



o'k n — k 

Moreover, equality holds in (|2.ip or (|2,2p at A if and only if A = c(l, 1, • • • ,1). 

The Newton-MacLaurin inequalities play a very important role in proving geometric inequal- 
ities mentioned above. However, we will see that these inequalities are not precise enough to 
show our inequality (11.91) . 

Let W 1 = M + x S n_1 with the hyperbolic metric 

g = dr 2 + sinh 2 rg Sn -i , 

where g§n-i is the standard round metric on the unit sphere S n_1 and £ C W 1 a smooth closed 
hypersurface in M n with a unit outward normal v. Let h be the second fundamental form of S 
and k = • • • , the set of principal curvatures of E in H n with respect to v. The /c-th 

mean curvature of £ is defined by 

o'k = o-k(n). 

We now consider the following curvature evolution equation 
(2.3) fx = Ff, 
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where = X(t, ■) is a family of hyper surf aces in HP, v is the unit outward normal to S t = X(t, •) 
and F is a speed function which may depend on the position vector X and principal curvatures 
of Y* t . One can check that along flow f|2 . 3[) . 



(2.4) j t j a k dn =(k + 1) J Fa k+1 dfi + {n - k) J Fa^dfi. 

For a proof see for instance [37] • Here we use a convention cr_i = 0. If one compares flow (|2.3p in 
HP with a similar flow of hypersurfaces in IP, the last term in (|2.4p is an extra term. This extra 
term comes from the sectional curvature —1 of HP and makes the phenomenon of hypersurfaces 
in HP much different from the one of hypersurfaces in BP. 

As mentioned above, we use exactly the following inverse flow 

(2.5) U = ^Z±^u. 
y ' dt 4 a 4 

By using the result of Gerhardt |23j, we have 

Proposition 2.2. If the initial hypersurface £ is horo 'spherical convex, then the solution for flow 
(|2.5p exists for all time t > and preserves the condition of horospherical convexity. Moreover, 
the hypersurfaces S t become more and more umbilical in the sense of 

\h) -S}\ < Ce~^, t>0, 

i.e., the principal curvatures are uniformly bounded and converge exponentially fast to one. Here 
hj = g lk hkj, where g is the induced metric and h is the second fundamental form. 

Proof. In [23j Gerhardt studied a more general inverse flow under a weaker condition that the 
initial surface is star-shaped. 

(2.6) j t X = -d>(i>, 

with a function $(r) = —r~ l for r > and F is a smooth curvature function, homogeneous of 
degree 1, monotone, and concave. What we only need to check is that flow ([2.5p or the general 
flow (12. 6|) preserves the condition of horospherical convexity. 

By using (4.23) in [23] for the second fundamental form /i*-, we have the evolution equation 

for h) :=h)- 8) that 



(2.7) 



where 



h) = Q(V 2 h, VKfj + §F kl h lr h r k h) + §F kl h lr h r k 5) 

+($ _ §F)h tk h kj + 2($ - + ($ - 6F)<5j 

-{($ + §F)5) - $F kl g kl h) - ^F kl g kl 5 i j }. 
:= Q{V 2 h,Vh)) + H), 

Q(V 2 h, Vh)) = §F kr h). kl + ^F kl ' rs h kl , jnrs ; + $(F kl h kl .j)( i F rs h r 



In order to use the maximum principle for symmetric tensors in [3j, which is a refinement of 
Hamilton's maximum principle |13j . we need to check the following two statements: 
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(i) H^a, > 0, 



H\a? ai = <S>(F kl h lr h r k + F kl g kl - 2F)\a\ 2 . 



(ii) N^maP + $sup r 2F kl {2T p k h ip ,ia i - r£l> pg ) > 0, 

for any a = (01, • • • , a n —i) with hjO? = 0, where a? = g^a;. 
From (|2.7p it is easy to say that 

Now, we can write 

for F is homogeneous of degree 1. Therefore, we infer in the orthonormal basis 

F kl h lr h r k + F kl g kl - 2F 

= F kl g ls (h s r + 5 s r )(hl + SI) + F kl g kl - 2F kl g ls (h s k + 5 s k ) 

= F kl g ls h s r h r k >0, 
since F kl is positive definite. On the other hand, we have 



F = F kl h kl , 



Hence, (i) follows. 
Noticing that 



$ > 0. 



Nj = ((fr(F)) kl ' rs h k i ; jh rs . i , 



and that the smallest eigenvalue of h equals to 1 , we can apply the work of Andrews [3] to show 
that statement (ii) holds. Hence the preservation of the horospherical convexity under (12.5H 
follows from his maximum principle. □ 

In [10], Cabezas-Rivas and Miquel showed that the preserving volume mean curvature flow 
preserves the horospherical convexity. See also the work of Makowski [35J . 

Let g be a Riemannian metric on M n ~ l . Denote Ric g and R g the Ricci tensor and the scalar 
curvature of g respectively. The Schouten tensor is defined by 

A„ = ( Ric„ 7— - — r 1 

9 n-3\ 9 2(n-2)' 

Let A g be the set of the eigenvalues of the Schouten tensor A g with respect to the metric g. The 
<7fc-scalar curvature, which is introduced by Viaclovsky, is defined by 

<rk(g) ■= cfc(A fl ). 

This is a natural generalization of the scalar curvature R. In fact, cr\(g) = 2 (n-2) -R- ecan that 
M is of dimension n — 1. We now consider the conformal class [<7§n-i] of the standard sphere 
S n_1 and the following functionals defined by 

n-l-2fc f 

(2.8) F k (g) = vol{g) »-i / a k {g)dg, k = 0, 1, n - 1. 



If a metric g satisfies (Jj(g) > for all j < k, we call it /c-positive and denote g G 1^ . We recall 
some generalized Sobolev inequality. 
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Proposition 2.3. Let < k < and g G [<7s«-i] k-positive. We have 

C k 2fc 

(2.9) Mg) > Mgsn-i) = -^Ci- 

Moreover, when k = 2, n > 5 and g £ [<7§™-i] 1-positive, the above inequality still holds. 



Inequality (|2.9p is a generalized Sobolev inequality, since when k = 1 inequality (|2.9p is just 
the optimal Sobolev inequality. See for example [1] and [12] . 

Proof. The first part follows from Theorem l.A in |29j . When k = 2, n > 5 and g G [<?§™-i] 
1-positive, by Theorem 1 in [19] (see also [26]), we infer 

*iG*)d<?) - 3 / ^( 5 )^>(— -) »-s^ £ LuZzl 

■1 ./sn-l ^ 8 



On the other hand, since g is 1-positive, we have 



n — 1 2 

V-l ' 



Hence, the desired result yields. □ 

3. An Alexsandrov-Fenchel inequality in the hyperbolic space 
First, a direct computation gives the following result. 
Lemma 3.1. Along the inverse flow \2. 5\) . we have 

d f( (n-3)(n-4) (n - l)(n - 2)(n - 3)(n - 4) 

"77 7 ^ °4 7 0"2 + 



tft is I 6 24 

/• f (n-3)(n-4) (n - l)(n - 2)(n - 3)(n - 4) 

(n - 5) / s r 4 e — CT2+ 21 

n-A{ f / gfegg 4(n-5) \ (n - 4)(n - 5) / 4(n - 3) _ ^ 



3.1 +^ / 5—-^ ^4 +- ^ 7^^2-3 

(n - 2)(ra - 3)(ra - 4)(n - 5) / <ri<7 3 _ 4(n - 1) 
24 \al n - 4 

Proo/. Under the flow (|23]1 . (l2T3j> yields that 



(n-3)(n-4) (n - l)(n - 2)(n - 3)(n - 4) 
a4 6 ° 2 + 24 



(n-4)(n-5) (n - 2)(n - 3)(n - 4)(n - 5) 
5cr 5 F 0-3F H — cri F. 

Substituting F = i nto the previous formula and arranging it, we thus get the desired 

result (ED). □ 



Remark 3.2. When n = 5, Lemma 3.1 implies that J-^fo is a constant. This is acturally the 
fact that J s I2 is the Euler characteristic of S up to a constant multiple. 
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Compared the last three terms in (|3.ip with the Newton-MacLaurin inequalities (|2.ip . (|2.2p . 
one will see immediately that the first two terms are non-positive, but the last one is non- 
negative. Therefore, unlike in the papers of [8], [15] and |34j we can not use the Newton- 
MacLaurin inequalities directly to establish the desired inequalities. We have to use more 
precise inequalities, which are fortunately true for any 

(3.2) K £ {k = K2, ■ ■ • , K n -l) € M"- 1 | Ki > 1}. 

This is one of key points of this paper. 

Proposition 3.3. Let n > 5. For any k satisfying ( 13. gj) we have a refined Newton-MacLaurin 
inequality 

a 5 a 3 4(n - 5) \ (n - 4)(n - 5) f 4(n - 3) a 2 

5 — ct 4 H t t °"2 - 3— 

o"4 n — 4 y b \ n — 4 04 

f33) (n-2)(n-3)(n- 4)(n - 5) Ar^ 4(n-l) \ 

1 ' J 24 V cr 4 n - 4 y ~ ' 

Equality holds if and only if one of the following two cases holds 

(3.4) either (i) Vi,j, or (ii)3i with «j > l&Kj = IV7 ^ i. 
Proof. For simplicity of notation, we denote 

(3.5) p k 



°n-l 



By a direct computation, it is easy to see that (|3,3p is equivalent to 
(3.6) - *)+ 2 L - #) + ( ™ - l) < 0. 



This inequality follows directly from the following two claims. 

Claim 1. 3(p2P4 — p|) + (P3P1 — Pi) < 0. Equality holds if and only if k satisfies ([3 
Claim 2. 3(^5^3 — pf) + (P3P1 — Pi) < 0. Equality holds if and only if k satisfies ([3 



In the proof of these two claims, we replace n — 1 by n for simplicity of notation and consider 
k = («i, «2> " " j K n) G ^ n with 

«i > 1, Vi, 

and denote the average of n-choose-/c type products for n real numbers «i, • • ■ ,K n with Kj > 1 
(l<J<n). 
Let 

(3.7) F n {x) = x n + C\p x x n - X + C 2 n p 2 x n - 2 + ■■■ + ^Pn-ix+Pn = n? =1 (s + 
which has exactly n real roots — < —1. By the mean value theorem we have that 
(3.8) 

-F^x) = x n ~ l + ^Icip^- 2 + —C 2 p 2 x n ~ 3 + ■■■ + icrV-i = ^-/(x + 
n n n n 
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is a (n — l)-degree polynomial with n — \ real roots — h L < — 1 (1 < i < n — 1). We can write 
(3.9) 

ii^(x) = X™- 1 + C l n _ lPl X n - 2 + Cl_ lP2 X n - 3 + ■■■ + C n n Z\p n -2X + P n-1 = + /*). 

This means that pi(l < i < n — 1) of k G IR n can be viewed as the average of (n — l)-choose-i 
type products of k = (hi, ■ ■ ■ , € M n_1 with kj > 1 for all j. Namely 

Pi(n) = Pi(k), for 1 < i < n — 1. 

Therefore, by an argument of mathematical induction, it suffices to prove Claim 1 for n = 4, 
and prove Claim 2 for n = 5. 

Proof of Claim 1 for n = 4: Given n numbers (k±, k 2 , • • • , K n ), we denote Yl f{ K i, " " " ■> K n) the 

eye 

cyclic summation which takes over all different terms of the type f(K±, • • • , K n ). For instance, 

n 

^Kl = Kl + K 2 H hBn, 5Z K 1 K2 = X] (^X^)' 

eye eye i=l j^j 

n 



cj/c i=l ^ l<i<fc<n ' 

= (n — 2) ki^2 — 6 K1K2K3. 



eye eye 

When n = 4, we have 



4^ A ' " 6^ x " "° 4 

eye eye C1/C 



Then we can get 

PlP3 - PA =^ + K 2 + K 3 + K 4 )(kiK 2 K3 + K1K2K4 + K1K3K4 + K 2 K 3 K 4 ) - 16kiK 2 K 3 K 4 ^ 

(3.10) =^^«ik 2 (k 3 - k 4 ) 2 , 

eye 

3(P2P4 -P\) =3^KlK2K 3 K 4 («;iK2 + Kl^ 3 + K 2 K 3 + K1K4 + K 2 K 4 + 
- ^(KlK 2 K 3 + KiK 2 K 4 + K1K3K4 + K 2 K 3 K 4 ) 2 ^ 

(3.11) =- Y^^2 K l K 2( K 3 -K 4 ) 2 , 



eye 

from above, we can infer that 



(3.12) 3(p 2 p 4 -p\) +P1P3 ~Pa = j£ ^^iK 2 (l - kik 2 )(k 3 - k 4 ) 2 < 0. 

This proves Claim 1. 



16 

eye 
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Proof of Claim 2 for n = 5: We have: 



Pi = g y^Kl, p 3 = — ^ KiK 2 K 3 , J> 4 = - ^«l« 2 «3«4 I P5 = «1«2«3«4«5- 
eye eye eye 

Hence 

1 5 1 

PlP3 - PA = T^(( K 1 + K 2 + K 3 + K 4 + K 5 ) V] K l K 2 K 3 - 10k 1 K 2 K 3 K 4: K 5 V] — 

eye i=l 

eye eye 

(3-13) =y^X] KiK2 ( K3 ~ K4 ) 2 ' 

eye 

3{P5P3 ~P\) =3^KlK2«3«4«5y^«l«2«3 ~ ^ ( K l"2«3«4) 2 

eye eye 

= — ( - 2 ^(kiK 2 K 3 K4) 2 + K l K 2 K 3 K 4 K 5 ^ KiK 2 K 3 

(3-14) =— ( - ^(KlK2K3) 2 (K4 - K 5 

eye 

Therefore, we have 

3(j>5P3 -pl) + (psPi - Pi) =-^J2 KlHi2 ( K3 ~ K ^ 2 + 100 ( ~ 5Z( KlK2K3 ) 2 ( K4 ~ Ks ) 

eye eye 

(3.15) =— ^ [«1K2 + ^2^3 + K1K3 - 3(kik 2 k 3 ) 2 ] (k 4 - K 5 ) 2 < 0, 

eye 

which implies Claim 2. Therefore the proof completes. □ 

Remark 3.4. One can also prove the Proposition by using a replacement with m = 1 + hi with 
hi > for all i and the ordinary Newtow- McLaughlin inequalities (Lemma \2. With the same 
proof we present here, Proposition 13.31 holds for k E M n ~ 1 with KiKj > 1 for all i 7^ j. This is 
equivalent to the condition that the sectional curvature o/S is non-negative. We believe that the 
results proved in this paper for horospherical convex hypersurfaces hold also for hypersurfaces 
with non-negative sectional curvature. 

Remark 3.5. From the proof of Proposition \3.3\ it is easy to see that (|3.3p changes sign for 
k G W 1 - 1 with < m < 1. 

Now we have a monotonicity of Q(T>t) defined by (jl.lip under flow (jl.lOjh 

Theorem 3.6. Functional Q(T, t ) is non-increasing under the flow (jl.lOp . provided that the 
initial surface is horospherical convex. 



eye cycle 

3 
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Proof. By Proposition 2.2, Lemma 3.1 and Proposition 13.31 we have 

d f( (n-3)(n-4) (n - l)(n - 2)(n - 3)(n - 4) 

"77 / ^ °"4 7^ 0"2 + 



d< 7e I 6 24 

/■/ (n-3)(n-4) (n - l)(n - 2)(n - 3)(n - 4) 
(3.16) < (re - 5) / < cr 4 <7 2 + 



E 



6 24 
On the other hand, by (|2.4p and (|2.2p . we also have 

(3.17) 7J7l^l= / — i — dM >(«-!) Sf 

Combining (|3.16p and ()3.17[) together, we complete the proof. 

□ 

Theorem 3.7. For any horospherical convex hypersurface E in H n urai/i n > 5, we have 

(3.18) QV> > (-')(-^-3)( B -4) ^ 
Equality holds if and only i/ E is a geodesic sphere. 

Proof. Let E(£) be a solution of flow (jl.lOp obtained by the work of Gerhardt [23]. We have 
showed that this flow preserves the horospherical convexity and non-increases the functional Q 
in proposition 2.2 and Theorem 13.61 respectively. Hence, to show (|3.18|) we only need to show 

(n-l)(n-2)(n -3)(n-4) _±_ 
^ v „, ._ 24 

Since E is a horospherical convex hypersurface in (M n , g), it can be written as a graph of function 
r(9), 6 G S n_1 . We denote X(t) as graphs r(t,8) on S"^ 1 with the standard metric g. We set 
A(r) = sinh(r) and we have A'(r) = cosh(r). It is clear that 

(A') 2 = (A) 2 + 1. 
We define <p(9) = <&(r(0)). Here $ is a function satisfying 

1 



(3.19) hmQ(E t )> — u n _ x . 



A 



We need another function 



v = Jl + \Vv\l 



'9 

By the result of Gerhardt [23], we have the following results. 
Lemma 3.8. 

A = 0(e^), \Vip\ + |VV| = 0(e - ^). 
The second fundamental form of E is written in an orthonormal basis 

i v\ [ i X' + v 2 X' J ■ 

We have also 

VA = XX'Vp. 
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We recall the basic facts 

1 



04 = — (s 1 - 65^2 + 8sis 3 + 3s 2 - 6s 4 ) , 



^2=2 (*i - s s) , 



where 



Sk 



From above, we have the following expansions. 

A^ 

vA 

( * 

v vA 



S2 



S3 



\ j V A' (A') 2 « 2 A' + 1 j ' 

W V A' (A') 2 + v 2 A' J + ^ l j ' 



v vA 
A' 



r^Vf 1 4Ay 6yfy> 4yVV^ A 



These give 
-2 l)(/;-2) 



A' 



and 



2(n-2)Ay yjyj /W 2 2(n-2) y y Wj \ 

(A') 2 A' « 2 A' y 

4(n - 2)A^ 6^ 



a4= 2i ( ^ )4(n " 3)(n " 4) U»-'W"- 2 ) 



A' 



A2 



(A') 



which imply 



(n-3)(n-4) (n - l)(n - 2)(n - 3)(n - 4) 

12 = 0"4 ~ C2 H 



6 

(n- l)(n-2)(n-3)(n-4) 
24 



24 



(n - 2)(n - 3)(n - 4) Aip ( A' V 
6 ^"Ua J 



<^> 2 - 1 



(n - 2)(n - 3)(n - 4) / A' 



6 



w 2 A' 



(n-3)(n-4) fAyA /V_ 

av V^a 

(„-:!)(/<- 1)^^A'_Y 



(-) 2 -- 
l fA ; 3 



(A') 2 W J 



(-) 2 -- 
W 3 



+ 0(e 
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From Lemma 13.81 and expressions of v, A, A', we get 

(^) 2 -l = ^-|V^| 2 + 0(e-^r), 



which implies 



6/2 (n-l)(n-2) 1 2 2 1 2 ^ 

{t7-\V<P\) -(ra-2) — (— - |V^| ). 



(n-3)(n-4) 4 V A 2 1 ^' y v A V A 2 



We now define a 2-tensor 
One can check that 



^ = -AVV-^(A 2 |V v | 2 -l)s. 

3^2 . -A , 1 



A"V 2 (r l A) + 0{e'—^). 



n-l 



(n-3)(n-4) 

Recall <p(0) = 3>(r(0)). It is easy to check that Aj = AVj. It follows that 
^ = VAA' and ^ = ^ - ^ + 0(e"^T). 

Hence we have 

, qo m 3Z 2 4 _ v V 2 A M®VA 1 |VA| 2 j« 

(3 ' 20) (n-3)(n-4) =A ^ ( "~ + " 2 ( ~ ~ + ° (c 

Recall the definition of the Schouten tensor 

n — 3 \ 2(n — 2) / 2 

Its conformal transformation formula is well-known (see for example |43J) 

N „ V 2 A 2VA (gi VA 1|VA[ 2 „ „ V 2 A 2VA <g> VA 1 |VA| 2 A 1 A 
(3.21) S A2 , = -— + — ^ --^ + S s = -— + — ^ j + 2 ff " 

From (I3T2T1 and (EOT]) , we obtain 

3/2 __ / , 2 



(r 2 (\ 2 g) + 0(e "-1). 



(n-3)(n-4) 

Recall that the metric on £(£) has the following expansion 

5 = X 2 (g + V</> ® V99) = X 2 g + 0(1). 

It follows 

7det(^) = A"" 1 (l + 0(e~^)), 

which gives 

|£(i)| = (1 + o(l)) / A"- 1 = (1 + o(l)M(A 2 5). 

Similarly, we have 

(3.22) f- ^ -=/ - 2 (A 2 5)^ A2§ + 0(e^) = (l + (l)) / a 2 (\ 2 g)dvol X 2g. 

J (n-3)(n-4) J§n-i y 7§„-i 
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(n— 5)t 

Here we have used the fact L n -i o- 2 (X g)dvol\2g = 0(e ) (in fact, we have Jg„-i o~2 (A g)dvol X 2^ > 

n-5 (n-5)t 

c[S(t)| ™- 1 > c\e for some c, ci > see the proof below). We try to use the generalized 
Sobolev inequality for the conformal metric X 2 g, which is presented in Proposition 2.3. For this 
purpose, we need to show that X 2 g £ r^~. Our observation is that it is asymptotically true. 
More precisely, we have the following asymptotic property 

a, = B _ 1+ 2zl(^_| Vv |»)-^ + 0( e -A) 

l.n-1 n-5|VAr AA, „,__«_, 

= n - 1 + <Ti(A^) +0(e 
Recall £(i) is a horospherical convex hypersurface. As a consequence, u\ > n — 1 so that 

o-i (A 2 ^) + 0(e~^r) > 0. 

t 

We consider A := A 1_e n ~ and the conformal metric X g. We have 

T2 /T2~\ n— 1 t_ n — 3 t_ <_,|VA| 2 ,_, ^n,2 m2m 

AVi(A'g) = — e ^— e (1 - e + (1 - e »-i)AVi(A 2 fl ). 

This yields A 2 g £ . From the Sobolev inequality (|2.9p for the 02 operator, we have 



(n-l)(n-2) 



(3.23) (vol{X 2 g))-—^ a 2 {X 2 g)dvol~ x2 > --u n _ x . 

JSn-l J O 

On the other hand, we have 

(3.24) (vol(X 2 g)r^ I o 2 (X 2 g)dvol~ x2 ~ = (1 + o(l))(vol(X 2 g)y^ [ a 2 (X 2 g)dvol X 2 § , 
since 

As a consequence of (|3.22p . (|3.23p and (|3.24[) . we deduce 

/ ,/ / nnx-"- 5 /" (n- l)(n-2)(n-3)(n-4) 
lira (wZ(E(t))) n-i / l 2 > S >A A ^ 

This proves (|3.19p . and hence (13.18p . When (|3.18j) is an equality, then Q is constant along the 
flow. In this case (13.17P is an equality, which implies that equality in the inequality 

n-4crio- 3 

— > n - 1, 

4 04 

holds. Therefore, £ is a geodesic sphere. □ 

Theorem 13.71 has its own interest. It is in fact a Sobolev type inequality. See similar Sobolev 
type inequalities in |29|. Now we can finish the proof of our main result. 



Proof of Theorem, 1.1. First of all, it is easy to check that for geodesic spheres all inequalities 
considered in this paper are equalities. 



16 



YUXIN GE, GUOFANG WANG, AND JIE WU 



In view of (jl.lip and (|3.18p . we have when n > 5, 

f In- l)(n-2)(n-3)(n-4) -±r .. , n- 5 
(3-25) / l 2 > K - ^ ^ ^ 



E 

Since we can write 04 = I2 + ( ra ~ 2 K w ~ 3 ) a2 _ (rc-i)(n-2)(>-3)(n-4) ^ com b mm g a bove with (jl.8p we 
get 



(3.26) J . 



> ^ f ,^i r ^ n g4 , f (n-2)(n-3) (n - l)(n - 2)(n - 3)(n - 4) 



6 24 



f 1 1 n-5 \ 2 

This is inequality (jl.9p . By Theorem 13.71 equality holds if and only if S is a geodesic sphere,. 

When n = 5, the Euler characteristic is equal to 1 since the hypersurface £ is star-shaped. 
By Remark 13.21 we know that (|3.25p is an equality when n = 5, even for any hypersurface 
diffeomorhpic to a sphere. Hence in this case, we also have the above inequalities with equality 
in (|3.26p . and hence (|1.9p . Equality in (jl.9p implies equality in (|3.27p . which, in turn, implies 
by [3l] that £ is a geodesic sphere. □ 
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